Syntactic heads in statistical language modeling

The use of syntactic structure in general and heads of syntactic constituents in particular has recently been shown to be beneecial for statistical language modeling. This paper provides an insightful analysis of this role of syntactic structure. It is shown that the predictive power of syntactic heads is mostly complementary to the predictive power of N-grams: they help in positions where an intervening phrase or clause separates the heads from the word being predicted, making the N-gram a poor predictor. Furthermore , a signiicant portion of this predictive power comes in the form of a more sophisticated back-oo eeect via the syntactic categories (nonterminal tags) of the heads. Finally, it is shown that using the categories of the syntactic heads is better than using the categories (part-of-speech tags) of the two preceding words, connrming that it is the syntactic analysis and not just the improved back-oo strategy which leads to improvements over N-gram models. Experimental results for perplexity and word error rate are presented on the Switchboard corpus to support this analysis.